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Abstract 

In the first part of this paper paper we show that a new technique exploiting ID correlation of 2D or even 
ID patches between successive frames may be sufficient to compute a satisfactory estimation of the optical 
flow field. The algorithm is well-suited to VLSI implementations. The sparse measurements provided by 
the technique can be used to compute qualitative properties of the flow for a number of different visual 
tasks. In particular, the second part of the paper shows how to combine our ID correlation technique with 
a scheme for detecting expansion or rotation ([5]) in a simple algorithm which also suggests interesting 
biological implications. The algorithm provides a rough estimate of time-to-crash. It was tested on real 
image sequences. We show its performance and compare the results to previous approaches. 
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1 Introduction 

The problem of how to compute efficiently estimates of 
the optical flow at sparse locations is of critical impor¬ 
tance for practical implementations in a number of differ¬ 
ent tasks. A specific example is the detection of expan¬ 
sion of the visual Held with a rough estimate of time-to- 
crash (TTC). The question has also interesting relations 
with biology, as we will discuss later. In this paper we 
propose an efficient algorithm for computing the opti¬ 
cal flow which performs well in a number of experiments 
with sequences of real images and is well suited to a VLSI 
implementation. 

Optical flow algorithms based on patchwise correla¬ 
tion of filtered images perform in a satisfactory way [3] 
and better in practice than most other approaches (see 
[1]). Their main drawback is computational complexity 
that forbid at present useful VLSI implementations. In 
this paper we show that ID patchwise correlation may 
provide a sufficiently accurate estimate of the optical 
flow 1 . We will then show with experiments on real im¬ 
age sequences how to apply this technique to measure 
time-to-crash, by exploiting a recently proposed scheme 
[5]. The latter scheme, which is robust and invariant 
to the position of the focus of expansion or the center 
of rotation, relies on sparse measurements of either the 
normal or the tangential component of the optical flow 
(relative to a closed contour). We will also discuss some 
broad implications of this work for the practical compu¬ 
tation of the optical flow and for biology, in particular 
its relation to Reichardt’s-type models. 

There are two main and quite separate contributions 
in this paper: 

1. an efficient ID correlation scheme to estimate the 
optical flow along a desired direction 

2 . the experimental demonstration that a previously 
proposed algorithm for estimating time-to-crash 
performs satisfactorily in a series of experiments 
with real images in which the elementary measure¬ 
ments of the flow are obtained by the new ID cor¬ 
relation scheme. 

2 Computing the Optical Flow along a 
Direction 

How can the component of the optical flow be measured 
efficiently along a certain desired direction? As argued 
by Verri and Poggio [12] a qualitative estimate is often 
sufficient for many visual tasks. For the task of detect¬ 
ing a potential crash, for instance, it has been suggested 
([5]) that a precise measurement of the normal compo¬ 
nent of the flow may not be necessary, since the precise 

1 In this paper we use mainly the L 2 distance rather than 
the correlation itself. Since the two measures are equivalent 
for the purposes of this paper, we will often use the terms 
“correlation” and “distance” in an interchangeable way. 


definition of the optical flow is itself somewhat arbitrary: 
it is sufficient that the estimate be qualitatively consis¬ 
tent with the values of the perspective 2D projection 
of the “true” 3D velocity field for ihe particular stim¬ 
ulus. In other words, even estimates that don’t really 
measure image-plane velocity (like Reichardt’s correla¬ 
tion model or equivalent energy models), since they also 
depend on spatial structure of the image, may be ac¬ 
ceptable for several visual tasks, if their estimates are 
consistent over the visual field. Certain uses of a crash 
detector are good examples. It turns out that even a 
rough estimate of time-to-crash (TTC) is possible using 
approximate estimates of the optical flow field. Flies and 
other insects rely for landing on what appears to be a 
qualitative estimate of the time-to-crash! 

2.1 ID correlation of 2D patches 

A possible approach for an approximative estimate of 
the optical flow is to use a ID correlation scheme be¬ 
tween two successive frames, instead of 2D correlation, 
as in [3]. The basic idea underlying the full 2D correla¬ 
tion technique that we label 2D-2D in this paper 2 is to 
measure, for each desired location, the (x,y) shift that 
maximizes the correlation between 2D patches centered 
around the desired location in successive frames. The 
patchwise correlation between the image at time t and 
at time t + St is defined as 

$(8x, by, t) = I W ®I = J I w (£,ry,t)I(6x+£, 6y+r)-,t+6t)d£dr) 

(1) 

where I w (£, 77 ; t) is the image at time t windowed to the 
patch of interest and set to 0 outside it. The L 2 distance 
has very similar properties to the correlation measure 3 . 

In the context of this paper, minimizing the L 2 distance 
is exactly equivalent to maximizing the correlation (the 
observation is due to F. Girosi). As noticed before [3], 
the previous idea can be regarded as an approximation 
of a regularization solution to the problem of computing 
the optical flow 4 . Usually, one does not use grey values 
directly but rather some filtered version of the image, 
for instance through a Laplacian-of-a-Gaussian filter (see 
[3]), possibly at different resolutions. 

Let us call D(8 x ,8 y ) the L 2 distance between 2 
patches in 2 frames at location (x, y) as a function of the 
shift vector (8 x ,8 y ). The “winner-take-all” scheme finds 
s* = (8*, 8*) that minimizes D (or maximize the corre¬ 
lation function $(5^,^)) and assumes that the optical 

2 It is also called winner-take-all method. 

“The L 2 distance is in this case the square root of the 
sum of the squares of the differences between values of cor¬ 
responding pixels. Other “robust” distance metric may be 
used, such as the sum of absolute values. 

4 And in turn several definitions of the optical flow such 
as Horn and Schunk’s, can be shown to be approximations of 
the correlation technique [6]. 
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Figure 1: The search space for the ID —2D scheme used 
for the computation of i lu* x and y components of the 
optical flow. 

flow estimate is u* = s*/A t, where At is the interframe 
interval. 

It is natural to consider whether the component of 
u* along a given direction, for instance x, may be es¬ 
timated in a satisfactory way simply by computing the 
6x that minimizes D(6x,0), that is the patchwise cor¬ 
relation as a function of x shifts only. We have found 
in our experiments that ID correlation of a 2D patch 
provides estimates of 6x* that are very close to the es¬ 
timates obtained from the 2D-2D technique. We label 
this technique 1D-2D, since it involves one-dimensional 
correlations on 2D patches. 

If we combine horizontal and vertical motion detectors 
of our ID, winner-take-all type (see fig.l), we obtain an 
appealing scheme to estimate the optical flow field at 
one point. The optical flow in one point is the vector 
sum of the x and y components computed by using such 
motion detectors. The key aspect of this approach is its 
reduction of the complexity of the problem, while main¬ 
taining a good estimation of the flow field: a complete 
two-dimensional search required in the winner-take-all 
scheme [3] is reduced to two one-dimensional searches. 
Let us call v max the maximum velocity expected on the 
image plane. In [3] the search space size to scan is 
(2v max + l) 2 for each point; in our approach, its size 
is limited to 2(2v max + 1). 

2.2 ID correlation of ID patches 

So far we have discussed that ID correlation of 2D 
patches gives a satisfactory estimate of the optical flow 
between two successive frames, reducing the search space 
of corresponding points. This is equivalent to saying that 


min&z<!>(6x, 0 ) 

and 

min&y $( 0 , 6y) 

give a satisfactory estimate of 

min SX)Sy $(6x, 6y). 

This suggests a further simplification: instead of <3> ((5.u, 0) 
consider a projection on x of <f> (<5.u , 6y) obtained by some 
form of averaging operation on y, that is 

$ * hn 

5 

where h'> is a 2D filter such as a Gaussian elongated in 
the y direction and * stands for the convolution operator. 
By well known properties of the Gaussian function, h'> 
can always be written as 

h'> = h * h, 

where h are Gaussian functions of appropriate variance. 
Assuming that we can neglet the patch size in the defi¬ 
nition of $, we can write: 

$ * ho = (It * h) O (It+bt * h) (2) 

where I t = I(x,y,t). 

Thus, in the approximation of a large patch size, pro¬ 
jecting the correlation function is equivalent to appropri¬ 
ately filtering the two images before correlation. Since 
it is usually better to discount the average intensity as 
well as small gradients through a high-pass filtering op¬ 
eration, in order to estimate the ^-component of u, we 
just perform a Gaussian smoothing in the y direction, as 
shown in eq. 2, and then perform an additional convo¬ 
lution with the first or second derivative of a Gaussian 
function elongated in the x direction. Therefore the in¬ 
tensity function that is used in practice in the correlation 
operation is: 

It = (G ay (y)*I t )*G^(x) ( 3 ) 

where a x and a y define the receptive field of such an 
elementary motion detector. After this filtering step, it 
is sufficient to evaluate thff; maximum of the correlation 
function only on ID patches to obtain an estimate of the 
x component of the flow. The previous argument does 
not strictly apply to the L'> distance measure that we 
have used in our experiments. The very close similarity 
between correlation and distance, however, suggests a 
very similar behavior in both cases. We label this tech¬ 
nique the ID-ID scheme since it involves ID correlations 
of ID patches. 






Figure 2: A TTC detector consisting of elementary mo¬ 
tion detectors (see figure 1) at several locations along a 
closed contour. Each of the elementary motion detec¬ 
tors could be replaced by a single detector normal to the 
circle. 

3 A crash detector: the Green theorem 
scheme 

As described in [5] (see also [2]), the divergence of the 
optical field Vu(r, y) is a differential measure of the local 
expansion (Vu(r, y) = dU:C Q*' y 1 + dUy g'y' y 1 )■ For a linear 
field (i.e. u(x) = Ax), the divergence of u is the same 
everywhere. In the case of linear fields (and all fields can 
be approximated by linear fields close to the singularity), 
the integral of the divergence over an area is invariant 
with respect to the position of the center of expansion. 
Green’s theorems show that the integral over a surface 
patch S of the divergence of a field u is equal to the 
integral along the patch boundary of the component of 
the field which is normal to the boundary (u • n). In 
formula 

/ Vu(r, y)dxdy = / un d.l. (4) 

Js Jc 

Therefore, since for a linear field Vu = 2/r where t 
is the time to crash (TTC), a TTC detector that ex¬ 
ploits the Green theorem just needs to sum over a closed 
contour, say a circle, the normal component of the flow 
measured at n points along the contour. We assume 
that the task is to compute time to crash (TTC) for 
pure translational motion. Possibly the simplest TTC 
detector of this type, shown in figure 3, is composed of 
just 4 elementary motion detectors. In this case we have 
to sum the £-component of u for the horizontal detectors 
and the t/-component of u for the vertical ones, with the 


Figure 3: Time-to-crash detector that exploits Green 
theorem. 

correct sign. 

Due to the invariance with respect to the position of 
the focus of expansion (or contraction) we can in princi¬ 
ple arrange a certain number of them (see fig.4) on the 
image plane. Our simulations suggest that one detec¬ 
tor with a large radius (fig. 3) is better than several, 
“smaller” detectors (fig. 4) in situations in which the 
whole visual field expands, probably because of better 
numerical stability of the estimates. Of course, a “large” 
detector has a poorer spatial resolution and this may be 
a problem in some applications (but not ours). 

We have discussed so far schemes for detecting expan¬ 
sion. Similar arguments hold for rotation. The Green ’s 
theorem relevant to this case is usually called Stokes’ 
theorem and takes the form 

/ V A u(a-, y) ■ dS = / u • dr (5) 

Js Jc 

which says that the total flux of the differential measure 
of “rotationality” of the field V A u across the surface 
patch S is equal to the integral along the boundary C 
of the surface patch of the component of the field which 
is tangential to the boundary. As described in [5], each 
elementary detector evaluates the tangential flow com¬ 
ponent at the contour of the receptive field (see fig.5). 
In this case a detector has to compute the component of 
u along the tangential direction at the contour. 

4 Experimental results 

4.1 The 1D-2D scheme 

We have extensively tested our approach on real image 
sequences. Each sequence was acquired from a camera 






Figure 4: A possible arrangement of TTC detectors in 
the image plane that is not as efficient as a single TTC 
detector with greater radius but has higher spatial reso¬ 
lution. 



Figure 5: Motion detector that exploits Stokes’ theorem. 


mounted on mobile platform moving at constant velocity. 
In all experiments the movement of the vehicle was a 
forward translation along a straight trajectory. We have 
verified the results obtained from our 1D-2D approach 
with the standard winner-take-all {2D-2D) scheme [3] 

t^- 

Figure 9 shows the first, and last, image of a. sequence 
composed of 100 frames. Each image of the sequence is 
first, convolved with a. Gaussian filter having a = 0.5. In 
both the algorithms we have used v max = 9 and v = 20 
pixels, where v max is the maximum expected velocity of 
the points on the image plane and v is the ray of the 
patch used for the evaluation of <F. In other words, the 
correlation window used for the optical flow computation 
is 41 x 41 pixels and the search space used is 19 x 19 
by 2D-2D and 19 + 19 by ID-2D. Figures 10 shows the 
optical flows computed by the two methods using two 
successive images of the sequence. The position of the 
focus of expansion was computed by using the approach 
described in [11]. 

We have used the method described in [11] and [5] to 
verify the TTC estimation. To compute the TTC at a. 
point, by using the method in [11], we used an area, of 
81 x 81 pixels around that point.. The points were 10 
pixels apart.. To compute TTC by using the method de¬ 
scribed in [5], we used a. lattice of overlapping motion de¬ 
tectors. The distance between two points on the lattice 
was 10 pixels. Each detectors had a. receptive field of ray 
r = 40 pixels. In fig. 11, we compare the results obtained 
by using the 2D-2D estimation of the optical flow with 
the 1D-2D one, by using the two different, methods in the 
first, stage of the TTC. Performing a. linear best. fit. on the 
TTC measurements, we obtain a. slope of m = —1.036 
by using the optical flows computed by 2D-2D and the 
method described in [11], and m = —1.139 by using the 
optical flows computed by 1D-2D and the method de¬ 
scribed in [5]. Comparing the true TTC (straight, line 
in fig. 11) with the TTC measures obtained by using 
the second method, we estimate an absolute error in the 
mean of 2.63, with a. standard deviation of 3.35 frame 
unit. In terms of relative units, the error in the mean is 
5.7% with a. standard deviation of 6.1%. 

4.2 The ID-ID scheme 

In this section we compare the results obtained by us¬ 
ing 2D-2D and ID-ID. In both techniques, we have used 
v max = 9 and v = 20 pixels. In other words, the cor¬ 
relation window used for the optical flow 1 Estimation is 
41 x 41 pixels for 2D-2D and 41 pixels for ID-ID. In 
the filtering step we have used a y = 6 and a x = 3 pix¬ 
els for computing the ^’-component, of the optical flow. 
These values of a produce ;||.receptive field of an elemen¬ 
tary motion detector equals to that used by 2D-2D (1681 
pixels). The fig. (6) shows a. plot, of the 2D correlation 
function used in 2D-2D over a. 2D search space and a. 
2D integration area. Figures (7) and (8) show a. plot, of 
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2D Distance Function 



Figure 6: 2D distance function. The arrows indicate the 
position of the minimum. 


ID Distance Function on X direction 



ID correlation functions used by the ID-ID technique 
to estimate both components of the optical flow. In this 
case we used two ID search spaces (in x and y directions 
respectively) and a ID integration area. Notice that this 
approach is capable of computing a reliable estimation 
of the flow vectors, while reducing i lie complexity of the 
problem. 

Figures (13), (17), (21) show the first and the last im¬ 
age of three sequences acquired from a camera mounted 
on a mobile platform moving at constant velocity, along 
a straight trajectory. Figures (14), (18), (22) show the 
optical flows computed by the two methods, by using 
two successive frames of the sequences. The mean (con¬ 
tinuous line) and the standard deviation (dashed line) 
of th# error on the optical flow estimation is shown in 
figures (15), (19), (23). Figures (16), (20), (24) show 
the TTC estimation by using the two different methods. 
In each experiments, we have used only one TTC detec¬ 
tor, with receptive field of r = 80 pixel, composed by 32 
elementary motion detectors (see fig. (2)). 

5 Conclusions 

5.1 Extensions of the optical flow algorithm 

There are several directions in which we plan to improve 
and extend our scheme: 

• it may be possible to reduce further the number of 
sample points for D p (i.e. the number of shifts) by 
using techniques for learning from examples such 
as the RBF technique ([4]) to approximate D p (6x) 
as D 1? (6x) = ^ ~2c n G(x — t n ), and then find the 
minimum of D p in terms of the dynamical sys¬ 
tem dx/dt = —e^2,c n G'(x — t n ). An alternative 
strategy is to try to learn directly the function 


Figure 7: ID distance function computed on the x di¬ 
rection by using ID-ID. 


ID Distance Function on Y direction 



Figure 8: ID distance function computed on the y direc¬ 
tion by using ID-ID. 
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minD p (x) from the samples of D p , using a few ex¬ 
amples of D p “typical” for the specific situation. 
The conjecture is that the RBF technique may be 
able to learn the mapping minD p (x) from examples 
of functions of the same class (compare Poggio and 
Vetter, 1992). A similar idea is to try to learn how 
to sample the correlation function as a function of 
past sampled values. Again, the training examples 
would be functions of the same class. This would 
provide at each t an estimate of the most appropri¬ 
ate correlation shifts to try. 

• instead of simply measuring Dip- 1 , that is the 
distance between frame i and frame i — 1, we 
could measure in addition also Dip- 2 , Dip- 3 , ...and 
combine them in an estimate of the optical flow 
component by taking the average of Dip-i/At, 
Dip- 2 /‘ 2 At, Dip-^/SAt, etc. This technique may 
be improved further by using a Kalman filter. 

• the same basic scheme of figure 1 may be used to 
compute horizontal and vertical disparities among 
the two frames of a stereo pair. 

• confidence measures will be developed to further 
improve the performance of the technique. 

5.2 Biological implications of our ID technique 

Poggio et al. ([5]) conjectured that ’’the specific type of 
elementary motion detectors that are used to provide the 
estimate of the normal component of the flow is probably 
not critical. Radially oriented (for expansion and con¬ 
traction), two input elementary motion detectors such 
as the correlation model [8, 9, 7, 10] - or approximations 
of it are likely to be adequate. The critical property is 
that they should measure motion with the correct sign.” 
Our results confirming their conjecture (since they sug¬ 
gest that ID correlation (or L 2 distance estimation) are 
sufficient for an adequate estimate of qualitative proper¬ 
ties of the optical flow) have interesting implications for 
biology. Consider a 2D array of Reichardt’s detectors 
(for motion in the x direction) with spacing A* and also 
detectors with spacings 2A* etc. Take the sum of all 
detectors with the same spacing over a 2D patch. Per¬ 
form a winner-take-all operation on these sums. Select 
the set with optimal spacing as the one corresponding 
to the present estimate of optical flow. This scheme is 
analog in time but otherwise equivalent to the one we 
have implemented. In formulae 

^2(Ii(t) - I i+ k(t - At)) 2 

where At is the interframe interval in our implementa¬ 
tion and is the delay in Reichardt’s model 5 , k represents 

5 We have written here the quadratic version of Reichardt’s 
model; the same argument carries over to the standard model 
with multiplication: for the basic equivalence of the the 
quadratic and multiplication version see [7]) 


the shift in our computation of D and represents the sep¬ 
aration between the inputs to Reichardt’s modules, Ii(t) 
is the image value (in general spatially and temporally 
filtered) at location i and time t and the sum ^ i s taken 
over the 2D patch of detectors of the same type. 

Thus an array of Reichardt’s models with different 
spacings of the 2 inputs (in x) could be used in a plau¬ 
sible way to estimate the optical flow component along 
the direction of the two-inputs detectors. Notice that 
a plausible implementation in terms of Reichardt’s de¬ 
tectors of the 2D correlation based algorithm would be 
much harder, since it would effectively require detectors 
with all possible 2D spacings. This seems implausible 
and contrary to experimental evidence in the fly, where 
only a small number of separations and directions (as 
small as 3) seem present. 

The above description is equivalent to our ID-2D 
scheme and involves the summation over x and y 
“patches” of elementary ID motion detectors. In the 
fly this is plausible, given the known summation proper¬ 
ties of specific wide Held lobula plate cells 6 . Our ID-ID 
scheme on the other hand would require a summation 
over the x dimension only (in our example) but an ori¬ 
ented filtering of the image with receptive fields elon¬ 
gated in y before the elementary motion detectors. It 
is possible that this second scheme may be used in the 
fly by different summation cells with smaller receptive 
fields. It is also possible that the wide field lobula plate 
cells effectively implement a scheme between the 1D-2D 
and the ID-ID by using some oriented filtering before 
motion detection and limited y integration of the output 
of the elementary motion detectors. Similar considera¬ 
tions may apply to some of the motion selective cortical 
cells. 

5.3 The Time-to-Crash detector 

The TTC detector we have simulated is not the only 
possible scheme. Others are possible (see for instance 
[2]) that take into account more complex motions than 
just frontal approach to a horizontal surface. 

It is also conceivable that the scheme we suggest may 
be simplified even further in certain situations. For in¬ 
stance, it may be sufficient in the summation stage to use 
the value of the correlation for a fixed (and reasonable) 
shift - instead of an estimate of the optical flow, that is 
the shift that maximize correlation. This is equivalent 
to use directly the output of Reichardt’s correlation nets 
instead of using the result of a winner-take-all operation 
on a set of Reichardt’s nets with different spacings (or 
delays). 

Another related idea is to continuously adjust the cor¬ 
relation shifts in order to track as closely as possible the 
maximum of the correlation (or the minimum of the dis¬ 
tance): in this way it may be possible to reduce the com- 

6 The patch would be very large and would correspond to 
the receptive field of the cell, that is its integration domain 
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putation of the correlation to just a few shifts, especially 
if time-filtering techniques are also used. 

Acknowledgments: We are grateful to John Harris and 
Federico Girosi for many discussions and very useful 
comments. 
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Figure 9: (a) First and (b) last, image of the sequence. 
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Figure 10: An example of optical flows computed by using (a) 2D-2D, (b) ID-2D and (c) ID-ID. In most frame 
pairs in a sequence the three flows are much more similar to each other. 













Figure 14: An example of a flow field computed at one point in time in the above sequence , obtained by using (a) 
2D-2D and (b) II)-II). 



Figure 15: Mean (dotted line) and standard deviation (dashed line) of the error of the optical flow estimation. 
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Figure 16: TTC estimation by using one TTC detector, with receptive field of r = 80 and 32 elementary motion 
detectors. The slope of the true TTC, computed by using the optical flows obtained by 2D-2D, is m = —0.672. The 
slope of the straight line, computed by using the TTC measures obtained by ID-ID , is m = —0.64. A comparison 
of the TTC measures obtained by ID-ID with the true TTC yields a mean absolute error of 9.02, with a standard 
deviation of 9.54. The relative error in the mean is 10.79% with a standard deviation of 9.49%. In order to evaluate 
the error in the time to crash estimation, the following steps have been performed. The true time to crash was 
estimated from a linear best fit. of the TTC measures obtained by using the 2D-2D scheme for the optical flow 
estimation. The figures show the straight line that represents the theoretical behavior of the TTC. A linear best 
fit. of the TTC measures obtained by using the ID-ID scheme for the optical flow estimation was then performed 
in order to evaluate the slopes of the two straight, lines. The absolute and relative error between the “true” TTC 
and the one measured by the ID-ID scheme was then estimated. Let. us call t* the true TTC. The absolute error is 
E a = \t* — t\ and the the relative error is E r = |r* — r|/|r|. 



Figure 17: (a.) First, and (b) last, image of the sequence. 
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Points = 1 Ray = 80 N = 32 



Figure 20: TTC estimation by using one TTC detector, with receptive field of r = 80 and 32 elementary motion 
detectors. The slope of the true TTC, computed by using the optical flows obtained by 2D-2D, is m = —0.77. The 
slope of the straight line, computed by using the TTC measures obtained by ID-ID , is m = —0.83. Comparing 
the TTC measures obtained by ID-ID with the true TTC, we had a mean absolute error of 8.02, with a standard 
deviation of 8.97. With respect to the relative error we had a mean of 10.9% and a standard deviation of 9.72%. 
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Points = 1 Ray = 80 N = 32 



Figure 24: TTC estimation by using one TTC detector, with receptive field of r = 80 and 32 elementary motion 
detectors. The slope of the true TTC, computed by using the optical flows obtained by wta-SD, is m = —1.24. The 
slope of the straight line, computed by using the TTC measures obtained by ID-ID , is m = —1.14. Comparing 
the TTC measures obtained by ID-ID with the t.rufe TTC, we had a mean absolute error of 7.6, with a standard 
deviation of 7.9. With respect to the relative error we had a mean of 11.4% and a standard deviation of 10.3%. 
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